feat(schema): load 0.x datasets as 1.0 records (core tables and labels) - #307

Summary

load_dataset() (#304) now loads 0.x datasets too, best effort. V0Converter(data_root).convert() (t4_devkit/schema/v1/convert_v0.py) reads the 0.x table files and returns 1.0 table dicts. These then go through the same build, validation and shortcut fields as a 1.0 dataset. T4Tables.manifest is None for 0.x.

Only the tables are converted, in memory. Data files are not touched: each frame keeps its own file and frame_index is null. vehicle_state and the traffic-light links come in the next PR.

How the converter is written

• One class: the data root, the 0.x tables and shared lookups are attributes, and there is one convert_<table> method per table. Only the placeholders are a class constant (FALLBACKS).
• Every 1.0 field is written out explicitly instead of copying 0.x records with **. That's easier to review, and an unexpected 0.x key can't leak into a 1.0 record (where from_dict would fail on it).
• No copied mappings: it reuses existing definitions (DistortionModel.from_num_coefficients, the 0.x VisibilityLevel.from_value) and reads every 0.x table file present instead of keeping a table list.

Core tables

• sensor: one row per 0.x calibrated_sensor, keeping its token, so sample_data.calibrated_sensor_token → sensor_token is a rename. index is in file order.
  • Camera: size from sample_data (or from the first image if 0.x stored 0); distortion model from the number of coefficients (new DistortionModel.from_num_coefficients: 0 none, 4 equidistant, 5 plumb_bob, 8 rational_polynomial; anything else raises).
  • LiDAR, described from its first frame: PCD fields from the header; .bin fields from V0_BIN_FIELDS cut to num_pts_feats (from the .info file if there is one, else 5); source_channels from the .info sources.
  • Source LiDARs of a concatenation have no frames of their own in 0.x, so here they get the default 5-field .bin layout and 0 frames. The next PR fills them from the concatenated channel's .info files.
• vehicle: one row per 0.x log (name = 0.x log.vehicle, null if empty, deterministic token).
• log: map_token from the 0.x map's log_tokens.
• map: sha256 computed from the osm file, projector from map_projector_info.yaml. No row if the map file is missing.

V0_BIN_FIELDS moved from dataclass/pointcloud.py (#305) to common/io.py, because the schema can't import from dataclass (which imports the schema). Its fifth field is now named channel, the 1.0 name (0.x called it ring; .bin stores no names).

Placeholders for values 0.x never stored

V0Converter.FALLBACKS, documented in docs/schema/v1/index.md ("Loading 0.x Datasets"):

vehicle.model == "unknown" means the dimensions are placeholders and must not be used. Real vehicle values are deliberately kept out of this public repo.

Labels

• Label origin: annotation_type is manual unless 0.x sets automatic_annotation. Then it is auto, provenance = the autolabel_metadata model names, and confidence = the score when there is exactly one model.
• category: is_thing is false only for categories used by surface_ann and by no object_ann or instance (road, sky).
• instance: index null (no point-wise labels yet), map_element empty; "" references → null.
• surface_ann → object_ann with instance_token null and no attributes; bbox computed from the RLE mask with pycocotools.
• visibility: the 0.x aliases are mapped (v0-40 → none, v40-60 → partial, v60-80 → most, v80-100 → full). 0.x unavailable and unknown levels are dropped, and boxes using them get visibility_token: null.

Schema change

sample_annotation.visibility_token is now nullable (null = unknown); docs updated. Real data needs it: 59 boxes across two of three downloaded datasets have unavailable visibility.

Checked on real data

• j6gen2 dataset (7 sensors, ~4,000 sample_data): converted in 0.13 s.
• Three 0.x annotation datasets with intermediate artifacts: 2,361 / 2,779 / 0 boxes, ~0.1 s each. The semantic segmentation one lists its 8 source LiDARs in LIDAR_CONCAT.source_channels, from its .info files.

Tests

tests/schema/v1/test_convert_v0.py, on the 0.x sample dataset:

• It loads as 1.0 records, with the documented placeholders.
• Labels convert; surface_ann becomes an object_ann without instance, with its box computed from the mask.
• An autolabel becomes annotation_type: auto, with the model as provenance.
• Visibility aliases are mapped; unavailable is dropped and becomes a null visibility_token.

tests/schema/v1/test_loader.py: the "0.x not supported yet" test is removed.